Pessimistic Testing

We propose a new approach to testing conformance to a nondeterministic specification , in which testing proceeds only as long as increased test coverage is guaranteed. In testing that a system meets a nondeterministic specification [1], it is usually assumed that the system is fair to each transition of the specification (i.e., the system will make every possible nondeterministic choice if given enough opportunities). But in fact, some transitions might be unlikely or impossible for a given implementation. When this happens, common model-based testing practices (e.g., following a precomputed tour of the state space) often lead to wasted test cycles and poor test coverage. We propose an alternative approach, in which the tester uses a dynamically computed strategy that is guaranteed to eventually increase coverage; testing stops as soon as the system has a strategy to avoid further coverage. As usual, we cast the test problem as a game; here, it is conveniently represented as a (directed) hypergraph. A hypergraph is given by a set of vertices and a set of (hyper)edges. Each edge is given by a head vertex and a set of tail vertices; we say it is incident to its head. An edge is reachable iff all of its tail vertices are reachable, and a vertex is reachable iff one of its incident edges is reachable (as usual, taking the minimal solution). The rank of a reachable edge is the maximum of the ranks of its tail vertices (0 if the tail is empty), and the rank of a reachable vertex is one plus the minimum rank of its reachable incident edges. In the test context, the hypergraph vertices are system states, each hyperedge represents a possible test stimulus, the head of the hyperedge is the state in which the stimulus can be delivered, and the tail of the hyperedge gives the states to which the system is allowed to transition under the stimulus. To keep track of which states have been explored by the test, we add trivial edges (with empty tails) incident on each state (other than the initial state). When the system first visits a state, this incident edge is removed, " marking " the state. Thus, the test state consists of a hypergraph and a current state (an unmarked vertex of the hypergraph), and a move of the testing game consists of the tester choosing an edge incident on the current state and …